Foundations of Emergency Medicine: Impact of a Standardized, Open-access, Core Content Curriculum on In-Training Exam Scores.

Introduction: Learners frequently benefit from modalities such as small-group, case-based teaching and interactive didactic experiences rather than passive learning methods. These contemporary techniques are features of Foundations of Emergency Medicine (FoEM) curricula, and particularly the Foundations I (F1) course, which targets first-year resident (PGY-1) learners. The American Board of Emergency Medicine administers the in-training exam (ITE) that provides an annual assessment of EM-specific medical knowledge. We sought to assess the effect of F1 implementation on ITE scores. Methods: We retrospectively analyzed data from interns at four EM residency programs accredited by the Accreditation Council for Graduate Medical Education. We collected data in 2021. Participating sites were geographically diverse and included three- and four-year training formats. We collected data from interns two years before (control group) and two years after (intervention group) implementation of F1 at each site. Year of F1 implementation ranged from 2015-2018 at participating sites. We abstracted data using a standard form including program, ITE raw score, year of ITE administration, US Medical Licensing Exam Step 1 score, Step 2 Clinical Knowledge (CK) score, and gender. We performed univariable and multivariable linear regression to explore differences between intervention and control groups. Results: We collected data for 180 PGY-1s. Step 1 and Step 2 CK scores were significant predictors of ITE in univariable analyses (both with P < 0.001). After accounting for Step 1 and Step 2 CK scores, we did not find F1 implementation to be a significant predictor of ITE score, P = 0.83. Conclusion: Implementation of F1 curricula did not show significant changes in performance on the ITE after controlling for important variables.

Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports.

While there is data assessing the test performance of artificial intelligence (AI) chatbots, including the Generative Pre-trained Transformer 4.0 (GPT 4) chatbot (ChatGPT 4.0), there is scarce data on its diagnostic accuracy of clinical cases. We assessed the large language model (LLM), ChatGPT 4.0, on its ability to answer questions from the United States Medical Licensing Exam (USMLE) Step 2, as well as its ability to generate a differential diagnosis based on corresponding clinical vignettes from published case reports. A total of 109 Step 2 Clinical Knowledge (CK) practice questions were inputted into both ChatGPT 3.5 and ChatGPT 4.0, asking ChatGPT to pick the correct answer. Compared to its previous version, ChatGPT 3.5, we found improved accuracy of ChatGPT 4.0 when answering these questions, from 47.7 to 87.2% (p = 0.035) respectively. Utilizing the topics tested on Step 2 CK questions, we additionally found 63 corresponding published case report vignettes and asked ChatGPT 4.0 to come up with its top three differential diagnosis. ChatGPT 4.0 accurately created a shortlist of differential diagnoses in 74.6% of the 63 case reports (74.6%). We analyzed ChatGPT 4.0's confidence in its diagnosis by asking it to rank its top three differentials from most to least likely. Out of the 47 correct diagnoses, 33 were the first (70.2%) on the differential diagnosis list, 11 were second (23.4%), and three were third (6.4%). Our study shows the continued iterative improvement in ChatGPT's ability to answer standardized USMLE questions accurately and provides insights into ChatGPT's clinical diagnostic accuracy.

[The new medical licensing regulations (ÄApprO) : Status and expected challenges]. / Die neue Ärztliche Approbationsordnung ÄApprO : Aktueller Stand und zu erwartende Herausforderungen.

BackgroundThe amendment to the medical licensing regulations (ÄApprO) was decided at the federal level in the version of the "Master Plan for Medical Studies 2020" passed in 2017. In addition to the organizational effort involved in redesigning the curricular teaching, the expected costs associated with the implementation of the new licensing regulations due to the necessary additional time and, therefore, personnel expenditure are of particular importance. Taking into account the different forms of study and the 20% scope for study-design provided to the individual faculties, the process of transferring the teaching content to the new modules confronts us with an enormous organizational challenge.Significance of O&UDiseases of the musculoskeletal system are of particular medical, social and economic importance. Therefore, the training of future physicians in the field of orthopedics and traumatology must be taken into account. The visibility of the field of orthopedics and traumatology must not be lost with the introduction of the new medical licensing regulations (ÄApprO).ImplementationThe implementation of the new medical licensing regulations at German universities will be costly and necessitates an increased number of staff. However, there is a great opportunity to position orthopedics and traumatology as a "central player" in the modular, interdisciplinary and interprofessional course landscape. It is, therefore, important to take on concrete responsibility for the design of the new teaching programs and to bring in our specialist and interdisciplinary skills wherever sensible and possible.

The Associations Between United States Medical Licensing Examination Performance and Outcomes of Patient Care.

PURPOSE: The United States Medical Licensing Examination (USMLE) comprises a series of assessments required for the licensure of U.S. MD-trained graduates as well as those who are trained internationally. Demonstration of a relationship between these examinations and outcomes of care is desirable for a process seeking to provide patients with safe and effective health care. METHOD: This was a retrospective cohort study of 196,881 hospitalizations in Pennsylvania over a 3-year period (January 1, 2017 to December 31, 2019) for 5 primary diagnoses: heart failure, acute myocardial infarction, stroke, pneumonia, or chronic obstructive pulmonary disease. The 1,765 attending physicians for these hospitalizations self-identified as family physicians or general internists. A converted score based on USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores was available, and the outcome measures were in-hospital mortality and log length of stay (LOS). The research team controlled for characteristics of patients, hospitals, and physicians. RESULTS: For in-hospital mortality, the adjusted odds ratio was 0.94 (95% confidence interval [CI] = 0.90, 0.99; P < .02). Each standard deviation increase in the converted score was associated with a 5.51% reduction in the odds of in-hospital mortality. For log LOS, the adjusted estimate was 0.99 (95% CI = 0.98, 0.99; P < .001). Each standard deviation increase in the converted score was associated with a 1.34% reduction in log LOS. CONCLUSIONS: Better provider USMLE performance was associated with lower in-hospital mortality and shorter log LOS for patients, although the magnitude of the latter is unlikely to be of practical significance. These findings add to the body of evidence that examines the validity of the USMLE licensure program.

Expiration of State Licensure Waivers and Out-of-State Telemedicine Relationships.

This cross-sectional study compares the use of telemedicine in states where COVID-19 pandemic­related licensure waivers expired vs states where waivers continued.

Reimagining a pass/fail clinical core clerkship: a US residency program director survey and meta-analysis.

Pass/fail (P/F) grading has emerged as an alternative to tiered clerkship grading. Systematically evaluating existing literature and surveying program directors (PD) perspectives on these consequential changes can guide educators in addressing inequalities in academia and students aiming to improve their residency applications. In our survey, a total of 1578 unique PD responses (63.1%) were obtained across 29 medical specialties. With the changes to United States Medical Licensure Examination (USMLE), responses showed increased importance of core clerkships with the implementation of Step 2CK cutoffs. PDs believed core clerkship performance was a reliable representation of an applicant's preparedness for residency, particularly in Accreditation Council for Graduate Medical Education's (ACGME)Medical Knowledge and Patient Care and Procedural Skills. PDs disagreed with P/F core clerkships because it more difficult to objectively compare applicants. No statistically significant differences in responses were found in PD preferential selection when comparing applicants from tiered and P/F core clerkship grading systems. If core clerkships adopted P/F scoring, PDs would further increase emphasis on narrative assessment, sub-internship evaluation, reference letters, academic awards, professional development and medical school prestige. In the meta-analysis, of 6 studies from 2,118 participants, adjusted scaled scores with mean difference from an equal variance model from PDs showed residents from tiered clerkship grading systems overall performance, learning ability, work habits, personal evaluations, residency selection and educational evaluation were not statistically significantly different than from residents from P/F systems. Overall, our dual study suggests that while PDs do not favor P/F core clerkships, PDs do not have a selection preference and do not report a difference in performance between applicants from P/F vs. tiered grading core clerkship systems, thus providing fertile grounds for institutions to examine the feasibility of adopting P/F grading for core clerkships.

Review of Complaints to the Rhode Island Board of Medical Licensure and Discipline 2018-2020.

OBJECTIVE: This study aimed to examine the patterns of complaints filed against physicians in Rhode Island, investigate the factors associated with complaint rates and outcomes, and assess the impact of the implementation of a new Framework for Just Culture. METHODS: Complaint data from the Rhode Island Department of Health's complaint tracker and physician licensing database were analyzed for the period of 2018 to 2020. Descriptive and statistical process control analyses were conducted to assess complaint rates, investigation rates, and adverse outcomes. RESULTS: Over the three-year period, 1672 complaints were filed against Rhode Island physicians, with approximately 40% of complaints being opened for investigation. The implementation of the Framework for Just Culture coincided with a sustained decrease in the rate of complaints opened. Failure to meet the minimum standard of care was the most common allegation, and male physicians and those aged 40-50 were more likely to have complaints filed against them. CONCLUSIONS: The study highlights the importance of complaint investigations in upholding standards for medical licensure and clinical competence. The Framework for Just Culture may have influenced the investigation process, resulting in fewer investigations opened without compromising the identification of cases requiring disciplinary action. These findings provide insights into physician accountability and the need for ongoing monitoring and improvement in complaint handling systems.

USMLE step 1 and step 2 CK as indicators of resident performance.

BACKGROUND: The purpose of this systematic review was to (1) determine the scope of literature measuring USMLE Step 1 and Step 2 CK as predictors or indicators of quality resident performance across all medical specialties and (2) summarize the ability of Step 1 and Step 2 CK to predict quality resident performance, stratified by ACGME specialties, based on available literature. METHODS: This systematic review was designed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [16]. The original search strategy surveyed MEDLINE and was adapted to survey Cochrane Library and Embase. A study was deemed eligible if it provided all three of the following relevant information: (a) Step 1 or Step 2 CK as indicators for (b) resident outcomes in (c) any ACGME accredited specialty training program. RESULTS: A total of 1803 articles were screened from three separate databases. The 92 included studies were stratified by specialty, with Surgery (21.7% [20/92]), Emergency Medicine (13.0% [12/92]), Internal Medicine (10.9% [10/92]), and Orthopedic Surgery (8.7% [8/92]) being the most common. Common resident performance measures included ITE scores, board certification, ACGME milestone ratings, and program director evaluations. CONCLUSIONS: Further studies are imperative to discern the utility of Step 1 and Step 2 CK as predictors of resident performance and as tools for resident recruitment and selection. The results of this systematic review suggest that a scored Step 1 dated prior to January 2022 can be useful as a tool in a holistic review of future resident performance, and that Step 2 CK score performance may be an effective tool in the holistic review process. Given its inherent complexity, multiple tools across many assessment modalities are necessary to assess resident performance comprehensively and effectively.

Increasing Diversity in the Physician Workforce: Pathway Programs and Predictive Analytics.

PROBLEM: Lack of diversity in the physician workforce has well-documented negative impacts on health outcomes. Evidence supports the use of pathway or pipeline programs to recruit underrepresented in medicine students. However, data on how a pathway program should deliver instruction are lacking. This report describes a multiyear project to build such a system with the goal of increasing diversity within medical school cohorts and ultimately the physician workforce. APPROACH: In the 2015-2016 academic year, the Ponce Health Sciences University started a 3-phase project to create a data-driven medical school feeder system by coupling a pathway program with predictive analytics. Phase 1 launched the pathway program. Phase 2 developed and validated a predictive model that estimates United States Medical Licensing Examination (USMLE) Step 1 performance. Phase 3 is underway and focuses on adoption, implementation, and support. OUTCOMES: Data analysis compared 2 groups of students (pathway vs direct) across specific factors, including Medical College Admission Test (MCAT) score, undergraduate grade point average (GPA), first-generation status, and Step 1 exam performance. Statistically significant differences were found between the 2 groups on the MCAT exam and undergraduate GPA; however, no significant differences were found between groups for first-generation status and performance on the Step 1 exam. This finding supports the authors' hypothesis that although pathway students have significantly lower mean MCAT exam scores compared with direct students, pathway students perform just as well on the USMLE Step 1 exam. NEXT STEPS: Next steps include expanding the project to another campus, adding more socioeconomic status and first-generation data, and identifying best curricular predictors. The authors recommend that medical school programs use pathway programs and predictive analytics to create a more data-centered approach to accepting students with the goal of increasing physician workforce diversity.

Demise of the USMLE Step-2 CS exam: Rationalizing a way forward.

The COVID-19 pandemic has compelled rethinking and changes in medical education, the most controversial perhaps being the cancelation of USMLE Step-2 Clinical Skills exam (Step-2 CS). What started in March of 2020 as suspension of this professional licensure exam, because of concerns about infection risk for examinees, standardized patients (SPs), and administrators, soon became permanent cancelation in January 2021. Expectedly, it triggered debate in medical education circles. Positively, however, the USMLE regulatory agencies (NBME and FSMB) saw an opportunity to innovate an exam tainted with perceptions of validity deficits, cost, examinee inconvenience, and worries about future pandemics; they therefore called for a public debate to fashion a way forward. We have approached the issue by defining Clinical Skills (CS), exploring its epistemology and historic evolution, including assessment modalities from Hippocratic times to the modern era. We defined CS as the art of medicine manifest in the physician-patient encounter as history taking (driven by communication skills and cultural competence) and physical examination. We classified CS components into knowledge and psychomotor skill domains, established their relative importance in the physician process (clinical reasoning) of diagnosis, thus establishing a theoretical framework for developing valid, reliable, feasible, fair, and verifiable CS assessment. Given the concerns for COVID-19 and future pandemics, we established that CS can largely be assessed remotely, and what could not, can be assessed locally (school/regional consortia level) as part of a USMLE-regulated/supervised assessment regimen with established national standards, thus maintaining USMLE's fiduciary responsibilities. We have suggested a national/regional program for faculty development in CS curriculum development, and assessment, including standard setting skills. This pool of expert faculty will form the nucleus of our proposed USMLE-regulated External Peer Review Initiative (EPRI). Finally, we suggest that CS evolves into an academic discipline/department of its own, rooted in scholarship.

Gearing Up for the U.S. Medical Licensing Examination Step 1 Pass/Fail Transition, Evaluating the Role of Research Productivity as an Alternative During the Neurosurgical Residency Selection Process: An Overview.

Neurosurgery residency programs are the most competitive among the surgical specialties for applicants to match into. U.S. Medical Licensing Examination (USMLE) step 1 scores are staple and pivotal in narrowing down applicants for granting interviews and ranking in the match process. The upcoming transition from a numeric to binary step 1 scoring shifts the emphasis on using the status quo Step 2 Clinical Knowledge numeric scoring as the only objective measure. It can prove insubstantial for accurate assessment of competencies, further urging residency programs to widen their selection protocols to allow multiple assessments of competency. Research experience has consistently proved to result in higher successful match rates, and a positive correlation is seen with the number of research publications and Hirsch indices. However, with a predicted emphasis on research with the shift in scoring practices, these tools provide inadequate insight into authorship, contributions, type of publications, and community impact, warranting the need to include supplementary modifications, surrogates, or alternatives to such tools for a more comprehensive and equitable assessment of research. This study summarizes the role of research in the neurosurgical match process, describes nuances in research evaluation, and introduces novel Hirsch indices and additional strategies to address these nuances for equitable evaluation of research productions.

Taking More Society for Academic Emergency Medicine Practice Tests Does Not Lead to Improved National EM-M4 Exam Scores.

INTRODUCTION: Emergency medicine (EM) is a required clerkship for third-year medical students, and an elective EM acting internship (AI) is available to fourth-year students at our institution. The Society for Academic Emergency Medicine's (SAEM) National Emergency Medicine M4 Examination (EM-M4) is administered to students at the end of the EM AI experience. To prepare for the exam, students gain access to 23 practice tests available from SAEM. In this study we investigate the correlation between the number of practice tests taken and EM-M4 performance. METHODS: We collected data for EM-M4 and the US Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) from students completing a MS4 EM clerkship in consecutive medical school classes from 2014-2017 at a private medical school. In addition, we collected data during the clerkship on the number of practice exams taken and whether a comprehensive practice exam was taken. We analyzed the study population three ways to determine whether the number of practice tests impacted final exam results: a binary distribution (1-11 or 12-23 tests taken); quaternary distribution (1-6, 7-12, 13-18, or 19-23 tests taken); and individual test variability (1,2,3,…22,23 tests taken). Complete data for 147 students was used for data analysis. RESULTS: The EM-M4 showed moderate (r = 0.49) correlations with USMLE Step 2 CK. There was no significant difference in EM-M4 performance in the binary analysis (P ≤ 0.09), the quaternary analysis (P ≤ 0.09), or the continuous variable analysis (P ≤ 0.52). Inclusion of a comprehensive practice test also did not correlate with EM-M4 performance (P ≤ 0.78). CONCLUSION: Degree of utilization of SAEM practice tests did not seem to correlate with performance on the EM-M4 examination at our institution. This could be due to many factors including that the question bank is composed of items that had poor item discrimination, possible inadequate coverage of EM curriculum, and/or use of alternative study methods. While further investigation is needed, if our conclusions prove generalizable, then using the SAEM practice tests is an extraneous cognitive load from a modality without proven benefit.

After the Discontinuation of Step 2 CS: A Collaborative Statement from the Directors of Clinical Skills Education (DOCS).

Issue: The United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), the only clinical skills competency testing required for licensure in the United States, has been discontinued. Evidence: This exam, though controversial, propelled a movement emphasizing the value of clinical skills instruction and assessment in undergraduate medical education. While disappointed by the loss of this national driver that facilitated standardization of clinical skills education, the Directors of Clinical Skills Education (DOCS) see prospects for educational innovation and growth. DOCS is a national organization and inclusive community of clinical skills education leaders. This statement from DOCS regarding the discontinuation of USMLE Step 2 CS has been informed by DOCS meetings, listserv discussions, an internal survey, and a review of recent literature. Implications: Rigorous clinical skills assessment remains central to effective and patient-centered healthcare. DOCS shares specific concerns as well as potential solutions. Now free from the external pressure to prepare students for success on Step 2 CS, clinical skills educators can reprioritize content and restructure clinical skills programs to best meet the needs of learners and the ever-evolving healthcare landscape. DOCS, as an organization of clinical skills leaders, makes the following recommendations: 1) Collaboration amongst institutions must be prioritized; clinical skills assessment consortia should be expanded. 2) Governing, accrediting, and licensing organizations should leverage their influence to support and require high quality clinical skills assessments. 3) UME clinical skills leaders should develop ways to identify students who perform with exceptional, borderline, and poor clinical skills at their local institutions. 4) UME leadership should fully commit resources and curricular time to graduate students with excellent clinical skills.